









Personal Infonration 
Managment Appications 1430 




AppBcation Speech Interface 
Ubrery 1420 




• 










POA Operating System 1440 




Serial Port Communlcatforw 
Interface 1410 




Personal Digital Assistant 1400 


i 


i 




t 


r 1 



Mtcnophone Button 1120 



[Microphorw. Amp& CODEC 113 



Speaker. Afnp.& COOEC 1140 



Han»mra "Cnglnof" (Corpcnont ♦ Orivtr) 
1100 



■ ▼ 




Serial Port CommufUcations 






Interlace 1030 




* 




Engine Manager 1020 


< 






Interpreter 1010 




^^^x^VKQblog Mafuow 1000 



Jser Interface File 
registration List 
1310 



TabtM 1300 



VocabularY 1321 1 



prompts 1322 



profile 1324 



Vocet>utarY 13ai] 



prompts 1332 | 



Command Reoognit>on 1210 



Vocabutary 1341 1 



prompts 1342 



SU Data 1320 



Launch Data 1330 Afp Data 1340 



I Text-to-Speech Conversion 1240 



Gpckan Languag* Efvokw* 1300 



31. J_ 




PDA 
2090 



Serial Port 2100 





\ 










Serial Port 




■ *w 




2060 






■ i^: 








: 

■ ?^ 












■ f - 

: ^ 










System 


: ^ 

* 




CPU 




Memory 


i 




2020 




2030 






i 








f 





System Bus 2010 



Microphone 
2070 



Microphone 

CODEC 

2040 



Speaker 
2080 



Speaker 
CODEC 
2050 



Mic, Vol. Reset 
buttons 4160 



I 



Button 

Interface 

4150 



Personal Speech Assistant 2000 



c z 



Personal Information 
Managment Appficstlons 3430 



i 



PCM Operating System 3440 



Appficatlon Speech Interface 
LGbrary 3420 



I 



Serial Port Communications 
Interface 3410 



Personal Digital Assistant 3400 



LEDs3150 



Loud Speaker 3140 



Microphone 3130 



Microphone Button 3120 



Power Managment 3110 



Hardware "Engines'* (Component + Driver) 
3100 




Convnunlcations Interface 
to Application Software 3030 



Engine Manager 3020 



interpreter 3010 



^^^^^^ ^Q[alog Mafiager 3000 



User Interface File 
registration List 
3310 



User Interface 
Files 3300 



Vocabulary 3321 



prompts 3322 



scripts 3323 



user profile 3324 



Vocabulary 3331 



prompts 3332 



scripts 3333 



profile 3334 




Convnand Recognition 3210 



User Verification 3220 



> Audio Recording and Playback 3230 



Text-to-Speech Conversion 3240 



Other Pluggable Engines 3250 



Spoken Language Engines 3200 



Vocabulary 3341 



prompts 3342 



scripte3343 



profile 3344 



SLI Data3320 



Launch Data3330 



App Data3340 



PDA 
4090 



Serial Port 4 100 



I 



Serial Port 
4060 



Auxiliary Comm 
Port 4 120 



X Power 

i Control 
=4090 



CPU 

4020 



System 

Memory 

4030 



I 



Microphone 
4070 



1 



External Mic 
Connector 4075 



Microphone 

CODEC 

4040 



Speaker 
4060 



1 



External Spkr 
Connector 4085 



Speaker 
CODEC 
4050 



I 





Mic. Vol. 
Reset 
buttons 
4160 


Non-Volatile 
Memory 4115 


; t 


Non-Volatile 
Memory 
Interface 
4110 


Button 

Interface 

4150 



Status 
Display 
4130 



I 



System Bus 401 0 



Personal Speech Assistant 4000 




sjn 



Earphone Connector 
5100 



External Microphone 
Connector 5110 



Volume 
Controls 
5080 



Micropone 

Button 

5070 




Reset 

Button- 

5090 



toud Speaker 

Enclosure 

5050 




LED State 
Indicators 
5060 



; Battery Mounting 
I Space 5040 



Spoken Language 
Interface Hardware 
5000 



PDA 
mounting 
space 
5010 



Serial Port Connector 
5020 



1 



Microphone 5120 



Screen 
5510 



Handwriting 
Area 5520 ^ 



Conventional PDA 
5500 



Scrolling 
Buttons 5540 




o o 



Shortcut Buttons 
5550 



Auxilliary Port Connector 5030 



GLOBALLY AVAIUBLE ELEMENTS 

[dialog^managerj 

root.voc 

root.pmt 

hardware^default.prf 

tapplicalionjaunchj 
launch.voc 

REGISTERED APPLICATIONS 

(address_book) 
addVappl.voc 
addVappl.pml 
addres.prf 

[memo_edilofJ 
memVappl.voc 
memVappl.pmt 

(date.bookj 

datVappl.voc 

datVappl.pmt 

dates.fsg 

Umes.fsg 



•Power Good- 
message 


► 


Extract Source 
and Message 


► 


Branch to 
Hardware 




Read 

Registration 




Load files listed 
in registraton 


Received 






System 
Code . 


► 


Flie into data 
structure 


► 


ftle into data 
structures 










s 








^09 


Sod 




80 








8Qb 





Bin 



mic_type 1 

enable_shutdown 0 

power Jime_out 120000 

shut_down_beep 1 

echo_commands 0 

prompt_on_not_recognized 1 
prompt_onJow_spoken_volume 1 

start_up_gender 0 

start_up_volume 85 

start_up_speed 75 

start_up_roughness 1 0 

start_up_breathlness 1 0 

start^up^headsize 40 

start_up_pitchfuctuation 50 

start-up jpilchbaseline 65 

start_up_quality 100 





User 

Depresses 


► 


Build 

Microphone 


► 


Interpret 
Microphone 




Send amended 
meseage to 




start Speech 
Decoder 




Microphone 
Button 




Button 
Message 




Button 
Message 


► 


Engine Managei 


► 




C 

7 

loooo 








<v 

J 




t OoOb 




LOOOB 




Speech 
Decoder 


► 


Interpreter 
Searches 
User Interlace 
Data for word 




Interpreter sends 
data assodater 
with word in tatilQ 
to Engine Manager 




Engine Manager 
fonnats data to 
target 

requirements 




Engine Managei 
sends formatted 


■ »=? 

: y 


returns word 








► 


► 


data to target 


ih 


s 




s 








S 




S 


• T • 

; 


IOC 10 




»oo to. 




(oot4- 








I ooiB 


i 5=: 


Speech 
Decoder 


► 


Interpreter 
Searches 
















i 


returns error 




User Inlefface 
Data for error 
messaoe ' 














; I ; 

i 


s 







































IS, 10 



new 


nw 


null 


scrolLup 


su 


10 


scioit^aown 


SO 




top 


to 


m til 

null 




DO 


null 


next 


nx 


null 


previous 


pr 


null 


ecJit_entry 


ed 


null 


done 


Ok 


null 


repeat 


rp 


1 


Iist_all_entries 


a! 


null 


Iist_all_name3 


an 


null 


address_please 


ad 


null 


address 


ad 


null 


whatjs_the_addres8 


ad 


null 


continue 


CO 


null 


stop_reading 


nop 


X 


find_homer_simpson 


fi 


Simpson 


find_Ed_Smith 


fi 


smith 


rind_David_Frank 


fi 


frank 




typedef struct 
{ 

char • commancljarget; 
char • spoken_command[CMDMAX]; 
char * command_string(CMDMAX]; 
char ' data_string[CMDMAX]; 
short arrayjength; 
} COMMAND.VOCAB; 
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typedef struct 
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' char • index_string[PROMPTMAX]; 
• . char • prompt(PROMPTMAX]; 
short array length; 
) PROMPT_t.lST; 



• € i 
i 1= 



Application 
requires data 


► 


Build 
prompt 


► 


find prompt 
Index in Ul 
data . 




construct 
prompt 
message 
string - 


to complete 




request 
Message * 




► 



I 3 COO 



I 



Send amended 
message to 
Engine Managei 



T 

iZOOh 



Start Text to 




Text to Speech 




Text to speech 


Speech 


► 


Converalon 




completed 


Conversion 




Completes 


► 


message 










returned 






s 




S 


(1.0 10 











SCRIPT Ul FILES ARE INDEX LISTS OF SCRIPTS 

verify_user python {verify} 

begin_dictation python {dictation} 

dialj)hone python {phone} 

greetings simple {greet} 

problems Inline say_"Houston._we_have_a_problemr,Shutdown; 



BOOL Verify_user() 
{ 

engine_manager.mic_button_means(user_verification_engine); 
lextJo_speech.say("Please verify your Identity by pressing the 

microphone button and speaking your name."); 
oJ< = verification_engine.verify_user(); 

engine_manager,mic_button_means(command_recognition_engine); 
relum(ok); 



Search 
built in 
commands 



iS'ooo 



Search 




Search 




Search 


active 




Application 




Dialog 


Vocabularies 




Launch 




Manager 


and Scripts 


— > 


Vocabulary 




Vocabulary 






and scripts 




and Scripts 






!> 









Search 
Background 
Application 
Vocabularies 



X 



Return 
Command 
Not Found 
Error 



P((S. IS 



Set current activity 
as •taking dictation" 



"1 



... Get date of last 
dictation activity 



Get stored count of 
total dictation activity 



compare total activit ^ 
to expert threshold 



1 6? 008 




compare time of last 
activity to threshold 

^tal > threshoid> H 
yes ^ 



I d IS ^ ' ^'^^ novice prompt 



. Play intermediate 


prompt 




\ 


7 


' Update activity 


record 





/6//7 



User 



Personal Speech Assistant 



App name placed in 
applist i^CQo 



Personal Digital Assistant 



User loads and starts new 
application on PDA \^ooo 




Application access Voice 
Library services |sh0(O 


— ► 



PSA searches application 
list for new app name i^o30 




TIew application loaded" 
PSA notification sent ( ^oOo 


< 








Application associated Ul 
Files requested ( ^ o4-0 


► 


Ul Flies extracted from 
applicaion resources ( 9oSo 










U (Files placed In data 
structures i:vo=K> 




Ul Flies formatted and 
transmitted I'^ofeO 


^ 



Application and data marked 
active i:»ot^o 



TTS speaks application 
name ready oo 



II In 









^ 7=- 


s 









J_ 



t>6^JV C6 



l'620O 



Fl(S. ISA 



Ad Hod Personal Area Network 



Personal Speech Assistant 



Personal Digital Assistant 



User enters area with 
services f ©boo 



Ul Files prepared and 
transmitted 1^04O 



Ad Hoc Network Discovery 
Protocol finds services IgOtO 



Request for Ut Hies sent to 
Ad Hoc Network (©OSO 



Application content sent to 
PDA te^OSO 



Application files not found 
(6020 



Application files installed 
lgGG?0 



Application Ul files Installed 
In data structures t90^o 



App name placed In 
app list \ <5oeQ 



Application and data marked 
active [QQCjO 



TTS speaks application 
name ready .(0 f 00 



Rs^ 18 B 



